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Unsupervised  Learning  from  Multiple  Information  Sources  Based  on  Non-negative  Matrix 

Factorization  (NMF) 

PI:  Tao  Li, 

School  of  Computing  and  Information  Sciences,  Florida  International  University 


Project  Objective 

In  many  real-world  applications,  the  data  are  naturally  multi-modal,  in  the  sense  that  they  are 
represented  by  multiple  sets  of  features.  For  example,  in  image  classification,  we  may  have 
features  directly  extracted  from  the  images  and  the  keywords  or  contextual  information  in  the 
environments.  For  target  identification,  we  may  have  images  taken  from  the  same  sensor  with 
different  viewpoints  or  images  taken  by  different  types  of  sensors.  In  general,  with  the 
availability  of  multiple  information  sources,  it  is  a  challenging  problem  to  conduct  integrated 
exploratory  analysis  with  the  aim  of  extracting  more  information  than  what  is  possible  from  only 
a  single  source. 

Nonnegative  matrix  factorization  (NMF)  factorizes  an  input  nonnegative  matrix  into  two 
nonnegative  matrices  of  lower  rank.  Recent  studies  show  that  NMF  provides  a  principled 
framework  for  unsupervised  learning.  In  this  proposal,  building  upon  the  foundations  of  NMF 
for  unsupervised  learning  that  the  PI  and  collaborators  discovered  recently,  aided  with  a 
substantial  body  of  work  by  PI  and  other  researchers,  we  propose  to  establish  a  comprehensive 
framework  for  unsupervised  learning  from  multiple  information  sources  based  on  NMF. 


Project  Approach 

The  proposed  framework  is  based  on  Nonnegative  matrix  factorization  (NMF).  It  has  been 
shown  that  NMF  with  the  sum  of  squared  error  cost  function  is  equivalent  to  a  relaxed  K-means 
clustering,  the  most  widely  unsupervised  learning  algorithm.  In  addition,  NMF  with  the  I- 
divergence  cost  function  is  equivalent  to  probabilistic  latent  semantic  indexing,  another 
unsupervised  learning  method  popularly  used  in  text  analysis.  These  results  established  the 
theoretical  foundation  for  NMF  to  solve  unsupervised  learning  problems. 

The  framework  has  several  advantages.  First,  NMF -based  approaches  are  amenable  to  vigorous 
analysis  and  benefits  from  the  well-established  knowledge  in  linear  algebra  accumulated  through 
centuries.  Second,  NMF-based  methods  can  be  efficiently  computed  thanks  to  the  mature 
software  packages  developed  by  scientific  computing  communities.  Third,  NMF-based 
algorithms  are  simple  to  implement  and  easy  to  understand,  comparing  with  the  probabilistic, 
information-theoretic  and  graph-theoretic  methods. 
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The  successful  developments  of  the  NMF-based  framework  utilize  the  established  computational 
mathematics  tools,  solve  challenging  data  fusion  problems,  and  have  a  variety  of  real-world 
applications.  This  new  framework  complements  the  existing  statistics  and  information  theory 
based  approaches. 


Scientific  Barriers 

The  first  challenge  is  on  extending  the  NMF  framework  from  one  data  source  to  multiple  data 
sources.  In  particular,  we  need  to  develop  NMF-based  algorithms  for  clustering  by  similarity 
fusion;  for  ensemble  clustering;  for  semi-supervised  clustering  from  multiple  information 
sources;  and  for  semi-supervised  learning  from  multiple  information  sources  with  limited 
supervision.  Another  challenge  is  how  to  scale  the  NMF-based  framework  for  large-scale 
datasets 


Scientific  Significance 


Fusing  multiple  information  sources  can  yield  significant  benefits  to  successfully  accomplish 
learning  tasks.  The  successful  developments  of  the  NMF-based  framework  utilize  the  established 
computational  mathematics  tools,  solve  challenging  data  fusion  problems,  and  have  a  variety  of 
real-world  applications  critical  to  the  Army’s  mission  such  as  automated  target  recognition  (e.g., 
for  smart  weapons),  guidance  for  autonomous  vehicles,  battlefield  surveillance,  and  automated 
threat  recognition.  The  NMF-based  framework  is  amenable  to  vigorous  analysis,  simple  to 
implement,  and  efficient.  This  framework  also  complements  the  existing  statistics  and 
information  theory  based  approaches. 


Scientific  Accomplishments 


In  traditional  soft  clustering,  a  data  point  may  be  assigned  to  multiple  clusters  and  the 
probabilities  of  a  point  belonging  to  different  clusters  are  directly  normalized.  In  [1],  we  show 
that  a  form  of  Nonnegative  Matrix  Factorization  (NMF)  based  clustering  has  the  nice  property 
that  posterior  cluster  probabilities  are  directly  normalized.  If  we  use  class  conditional  probability 
normalization,  the  NMF-based  reduces  to  Probabilistic  Latent  Semantic  Indexing  (PLSI).  We 
also  provide  a  NMF  based  framework  for  unifying  posterior  probabilistic  clustering  and 
probabilistic  latent  semantic  Indexing.  Figure  1  shows  an  example  for  illustrating  the  proposed 
posterior  NMF.  In  a  recent  book  chapter  [4],  we  provide  a  comprehensive  review  of  non¬ 
negative  matrix  factorization  methods  for  clustering.  In  particular,  we  outline  the  theoretical 
foundations  on  NMF  for  clustering,  provide  an  overview  of  different  variants  on  NMF 
formulations,  and  examine  several  practical  issues  in  NMF  algorithms.  We  also  summarize 
recent  advances  on  using  NMF-based  methods  for  solving  many  other  clustering  problems 
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including  co-clustering,  semi-supervised  elustering,  and  consensus  clustering  and  diseuss  some 
future  research  directions. 


Figure  1 ;  The  original  data,  shown  on  the  left,  eontains  two  clusters;  the  black  points  are  in  the 
middle  of  two  elusters.  Result  of  running  the  proposed  posterior  NMF  is  placed  in  the  middle. 

is  the  matrix  factor  obtained  from  the  proposed  NMF.  As  reference,  matrix  factor  F 

obtained  from  traditional  NMF  is  also  given  by  “1^”.  The  heat  map  of  G  of  the  posterior  NMF, 
shown  on  the  right,  indicates  that  the  black  points  belong  to  multiple  clusters  with  almost  similar 
degrees  of  membership  (best  viewed  in  eolor). 


The  team  has  developed  a  novel  multimedia  information  fusion  framework,  in  which  two  multi¬ 
view  elassifieation  approaches  -  (Dynamie  Weighting  and  Region-based  Semantie  Coneept 
Integration)  for  categorizing  the  images  under  the  “supervision”  of  topie-related  textual 
deseriptions,  are  seamlessly  integrated  by  analyzing  the  special  characteristics  of  different 
images  [2].  Notice  that,  the  proposed  framework  is  a  generic  multimedia  information  fusion 
framework  which  is  not  limited  to  the  aforementioned  approaches,  and  it  ean  also  be  used  to 
integrate  other  existing  multi-view  classification  methods  or  models.  Also,  our  proposed 
framework  is  capable  of  handling  the  large  scale  image  categorization.  Specifically,  the  proposed 
framework  can  automatically  choose  an  appropriate  classification  model  for  each  testing  image 
aeeording  to  its  special  characteristics  and  consequently  aehieve  better  elassifieation 
performance  with  relatively  less  computation  time  for  large  seale  datasets;  Moreover,  it  is  able  to 
categorize  images  without  any  textual  description  in  real  world  applications.  Empirical 
experiments  on  two  different  types  of  web  image  datasets  demonstrate  the  efficacy  and 
efficiency  of  our  proposed  elassifieation  framework.  An  overview  of  our  proposed  framework  is 
depieted  as  in  Figure  2. 

In  [3],  we  have  applied  our  developed  multimedia  fusion  framework  in  disaster  management  to 
semantically  associate  situation  reports  with  disaster-related  multimedia  data.  We  have 
designed  and  developed  a  Multimedia- Aided  Disaster  information  Integration  System  (MADIS) 
within  an  iPad-specific  application  that  conveys  all  such  information  via  a  unified  and  intuitive 
graphical  interface.  The  mobility  of  the  iPad  device  provides  the  EM  personnel  with  free  and  fast 
interaction  in  communicating  between  both  the  eommand  eenters  and  the  actual  disaster  sites. 
Eigure  2  shows  the  overview  of  the  MADIS  system. 
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Figure  2.  The  Multimedia  information  Fusion  Framework  for  Web  image  Categorization 
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Figure  3.  The  Overview  of  the  MADIS  System 
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In  [5],  we  propose  a  novel  eross-domain  learning  method  (shown  in  Figure  4)  to  elassify  these 
web  multimedia  objeets  by  transferring  the  eorrelation  knowledge  among  different  information 
sourees.  Here,  the  knowledge  is  extraeted  from  unlabeled  objeets  through  unsupervised  learning 
and  applied  to  perform  supervised  classification  tasks.  To  mine  more  meaningful  correlation 
knowledge,  instead  of  using  commonly  used  visual  words  in  the  traditional  bag-of-visual-words 
(BoW)  model,  we  discover  higher  level  visual  components  (words  and  phrases)  to  incorporate 
the  spatial  and  semantic  information  into  our  image  representation  model,  i.e.,  bag-of-visual- 
phrases  (BoP).  By  combining  the  enriched  visual  components  with  the  textual  words,  we 
calculate  the  frequently  co-occurring  pairs  among  them  to  construct  a  cross-domain  correlated 
graph  in  which  the  correlation  knowledge  is  mined.  After  that,  we  investigate  two  different 
strategies  to  apply  such  knowledge  to  enrich  the  feature  space  where  the  supervised  classification 
is  performed.  By  transferring  such  knowledge,  our  cross-domain  transfer  learning  method  can 
not  only  handle  large  scale  web  multimedia  objects,  but  also  deal  with  the  situation  that  the 
textual  descriptions  of  a  small  portion  of  web  images  are  missing.  In  [6],  we  introduce  a  novel 
framework  for  generating  pictorial  storylines  for  given  topics  from  text  and  image  data  on  the 
Internet.  Unlike  traditional  text  summarization  and  timeline  generation  systems,  the  proposed 
framework  combines  text  and  image  analysis  and  delivers  a  storyline  containing  textual,  pictorial, 
and  structural  information  to  provide  a  sketch  of  the  topic  evolution. 


Figure  4.  The  Framework  of  the  cross-domain  transfer 


Ensemble  clustering  refers  to  the  problem  of  combining  different  (input)  clusterings  of  a  given 
dataset  to  generate  a  final  (consensus)  clustering  that  is  a  better  fit  in  some  sense  than  existing 
clusterings.  Over  the  past  few  years,  many  ensemble  clustering  approaches  have  been  developed. 
However,  most  of  them  are  designed  for  partitional  clustering  methods,  and  few  research  efforts 


5 


Proposal  No.  56759-CS-H 

Project  Title:  Unsupervised  Learning  from  Multiple  Information  Sources  Based  on  Non-negative  Matrix  Factorization  (NMF) 
PI:  Tao  Li,  Florida  International  University 


have  been  reported  for  ensemble  hierarchieal  clustering  methods.  In  [7],  a  hierarchical  ensemble 
clustering  framework  that  can  naturally  combine  both  partitional  clustering  and  hierarchical 
clustering  results  is  proposed.  In  addition,  a  novel  method  for  learning  the  ultra-metric  distance 
from  the  aggregated  distance  matrices  and  generating  final  hierarchical  clustering  with  enhanced 
cluster  separation  is  developed  based  on  the  ultra-metric  distance  for  hierarchical  clustering. 

We  also  studied  some  other  related  problems  for  information  fusion.  In  [8],  we  proposed  an 
extensible  framework  -  META  -  to  enable  analysts  to  easily  and  selectively  extract  and 
summarize  events  from  different  views  with  different  resolutions.  In  [9],  we  explored  ensemble 
strategies  of  contextual  bandit  algorithms  to  obtain  robust  predicted  click-through  rate  (CTR)  of 
web  objects.  The  ensemble  is  acquired  by  aggregating  different  pulling  policies  of  bandit 
algorithms,  rather  than  forcing  the  agreement  of  prediction  results  or  learning  a  unified  predictive 
model. 

Collaborations  and  Leveraged  Funding 

One  student  supported  by  Florida  International  University  Presidential  Fellowship  and  one 
student  supported  by  China  Council  Scholarship  have  involved  in  the  project.  Recently,  we 
received  a  grant  from  Purdue  VACCINE  Center/Department  of  Homeland  Security  (DHS)  to 
develop  a  data  mining  framework  for  enhancing  emergency  response  situation  reports  with 
multi-agency,  multi-partner,  multimedia  data. 


Technology  Transfer 

We  have  been  collaborating  with  the  Emergency  Management  (EM)  personnel  at  Miami-Dade 
Emergency  Management  (MDEM)  and  they  showed  strong  interest  in  using  the  Multimedia- 
Aided  Disaster  information  Integration  System  to  perform  daily  operations.  We  are  encouraged 
to  further  develop  the  system  into  an  operational  pilot  and  promote  the  commercialization  of  the 
system  for  benefitting  the  whole  EM  community. 


Anticipated  Scientific  Accomplishments 

We  will  publish  our  research  results  in  reputable  conferences  and  journals.  In  addition,  we  will 
develop  a  versatile  and  user-friendly  tool  that  implements  our  developed  NMF  algorithms  in 
different  analysis  contexts  to  support  the  above-mentioned  applications.  The  program 
components  (as  source  code)  will  be  made  available  to  interested  researchers  through  the  project 
web  pages.  We  will  package  our  developed  toolkits  in  a  form  that  can  be  quickly  deployed  to  an 
interested  user. 
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